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MAXIMUM LIKELIHOOD SEQUENCE ESTIMATOR WHICH 
COMPUTES BRANCH METRICS IN REAL TIME 

CROSS-REFERENCES TO RELATED APPLICATIONS 
This application claims priority to provisional application serial number 
60/147,506, filed August 9, 1999. 

The present invention relates to maximum likelihood sequence estimators 
utilized by receivers in communications systems to mitigate intersymbol interference in 
digital data propagation caused by dispersive communication channels. 

BACKGROUND OF THE INVENTION 

A well known problem in communication systems is intersymbol 
interference induced on a digital communication waveform by the dispersive properties of 
a transmission channel. 

In most wireless communication systems, a transmitter will radiate its 
output signal in all directions. This radiated data can reflect off buildings, windows, and 
other surfaces. A receiver will therefore receive data from the transmitter which has 
traveled in a variety of routes. For example, an individual symbol of data may reach the 
receiver by traveling in a straight line, and may also reach it by reflecting from a building, 
and may further reach the receiver by first reflecting from a body of water, then off a 
different building. This will mean that the same symbol will travel form transmitter to 
receiver by three paths. This phenomena is referred to as multipath. 

The result of this multipath effect is that each symbol is in effect smudged 
in time, and each symbol sent by the transmitter blurs into adjacent symbols. Therefore, 
the received waveform at any given time is dependent on some number of previous 
symbols. This is known as intersymbol interference. Furthermore, this multipath effect, 
as well as other effects, are what causes dispersion in the channel. 

A class of equalizers, known as Maximum Likelihood Sequence 
Estimators (MLSE) has been developed to correct for this intersymbol interference. 
Many of this type of equalizer have incorporated what is known as the Viterbi Algorithm 
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for use in determining the most likely sequence. This Algorithm is however, very 
computationally intensive, and requires a great deal of memory which must be integrated 
onto a VLSI chip designed to include an MLSE. Embodiments of the present invention 
provide improvements to an MLSE resulting in reduced memory requirements, simpler 
5 computational complexity, and improved accuracy. These improvements lead to reduced 
power consumption, smaller die size for the VLSI circuit. 

SUMMARY OF THE INVENTION 
One embodiment of the present invention provides for real time 
10 computation of the branch metrics used in the partial path metric decisions made during 
the forward trace through the trellis. 

Another embodiment of the present invention provides for the use of Gray 
coding of states to reduce the computations required to calculate the branch metrics at 
each node of the trellis. 

1 5 A further embodiment of the present invention makes use of known initial 

and final states of the trellis to improve MLSE accuracy. An enhancement to this 
embodiment further uses knowledge of invalid bits in the initial and final states. 

BRIEF DESCRIPTION OF THE DRAWINGS 
20 Fig. 1 shows a data burst for the GSM (Global System for Mobile) 

standard; 

Fig. 2 shows the relationship between the transition bit, the survivor bit, 
and the present state of the channel; 

Fig. 3 shows the details between two consecutive stages of the trellis; 
25 Fig. 4 is a diagram of a trellis used by the MLSE; 

Fig. 5 is an example of the path metric comparisons; 
Fig. 6 is a further example of the path metric comparisons; 
Fig. 7 is a tabular example of the recursive method of computation of 
branch metric parameters F(q,0) and F(q,l) by the F-Generator according to the present 
30 invention; 

Fig. 8 is an example of a trellis having known valid initial and final states; 
Fig. 9 is a block diagram describing a general linear carrier modulated data 
transmission system; 

Fig. 1 0 is a block diagram of the ML receiver; 
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Fig. 1 1 is a block diagram showing an overview of the MLSE 
computational method according to the present invention; 

Fig. 12 is an illustration of the preferred embodiment for the F-Generator 
method of computation of branch metric parameters F(q,0) and F(q,l); 

Fig. 13 is a flow chart of a method of computation of branch metric 
parameters in accordance with the present invention; 

Fig. 14 describes an embodiment for the ALU method of Viterbi 
Algorithm calculation for the forward trace phase; 

Fig. 15 describes an embodiment for the ALU method of Viterbi 
Algorithm calculation for backward trace phase of operation; 

Fig. 16 describes modifications to the ALU of the preferred embodiment 
for incorporation of known initial state information in one embodiment; 

Fig. 17 describes modifications to the ALU of an embodiment for 
incorporation of known final state information; 

Fig. 18 describes an embodiment of the ALU for reduced computational 

complexity; 

Fig. 19 shows a flow chart for the Viterbi Algorithm used by one 
embodiment of the present invention which includes soft decision making; 

Fig. 20 shows a flow chart for one embodiment of the present invention 
which incorporates knowledge about the initial and final states; 

Table 1 shows the ALU and F-Generator processing steps for one 
embodiment of the present invention; 

Table 2 shows the required number of clock cycles needed by the ALU for 
one embodiment; 

Table 3 shows the memory requirement; 

Table 4 shows the register contents regarding the initial and final states of 
a trellis for the GSM standard; and 

Table 5 shows the ALU and F-Generator processing steps for another 
embodiment of the present invention. 

DESCRIPTION OF THE SPECIFIC EMBODIMENTS 
Embodiments of the present invention relate to improvements in MLSEs 
used in receivers to compensate for intersymbol interference caused by the dispersive 
nature of transmission channels. 
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Fig. 1 is a diagram of a data burst for the GSM standard. The data burst 5 
has two data sections, one on either side of a 26 bit midamble 10. The data sections each 
are 58 bits in length, and are referred to as the Left Hand Side (LHS) data 20, and the 
Right Hand Side (RHS) data 30. A series of dummy bits 22, and a tail of 3 zeros 25 
5 precede the LHS data 20. A tail of 3 zeros 35, and a series of dummy bits 32 follow the 
RHS data. The power ramps up and down while the dummy bits are sent, and the power 
must be constant to plus or minus ldB while the tails 25 and 35, midamble 10, RHS 30, 
and LHS 20 data are sent. 

Once the data burst has been received, the RHS 30 and LHS 20 data may be stored. The 
1 0 midamble 1 0 is processed to create a model for the channel characteristics that existed 
when the data burst was sent. The model is made up of a series of L coefficients s i5 s 2 . . 
.s L , which are autocorrelation values of the impulse response of the channel. These 
coefficients will be used below to calculate the branch metrics used in the trellis 
processing. 

15 Fig. 2 shows the relationship between the incoming data 210, the transition 

bit 220, the survivor bit 230, the present state 240, and the bits in a channel memory of 
length L 250. The incoming data 210 is the hypothesized transmitted sequence. This 
sequence will be processed by the MLSE to create the corrected received data sequence. 
As discussed above, each incoming data bit is smudged, or dispersed into the next several 

20 following bits. The length of the dispersion is referred to as the channel memory, which 
has a length of L. The incoming data 210 can be thought to be passed through a shift 
register 250 of length L. As data shifts from left to right, the newly incoming bit is the 
transition bit 220, and is referred to as the MSB, the outgoing bit is the survivor bit 230, 
and the bits in the shift register 250 form the present state q 240. It is important to note 

25 that each present state q 240 has two subsequent states which are possible, depending on 
whether the next transition bit 220 is a one or a zero. For example, a present state with a 
length L of 4 bits 0001 has two subsequent states possible; 0000 if the transition bit is a 0, 
and 1000 if the transition bit is a 1. In either case, the 1 in the LSB position of the present 
state is shifted out and becomes the survivor bit 230. Put another way, each present state 

30 q 240 has two predecessor states, depending on the survivor bit that was shifted out. For 
example, state 0000 has predecessor states 0000 and 0001, where the survivor bits are 0 
and 1 respectively. The survivor bit defines the predecessor state, and is alternately 
referred to as the predecessor bit. 
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Fig. 3 is a diagram showing all possible present states and associated 
subsequent states for a MLSE having a channel length of L=4. In Fig. 3 the MSB is on 
the right, and the LSB is on the left. As one example, present state number 0, which is 
the first row in column 310 has the present state of 0000. If the transition bit is a 0, then 
5 the path from the present state to the subsequent state follows branch 330, the survivor bit 
230 is a zero, and the subsequent state is 0. If the transition bit is a 1, then the path from 
the present state to the subsequent state follows branch 340, the survivor bit 230 is a zero, 
and the subsequent state is 0001 . Note that if the MSB is written on the left, this 
subsequent state is 1000. In this diagram, all 2X2 L , or 32 branches between the 2 L , or 

10 16 present states 310, and 2 L next states 350 are shown. 

Fig. 4 is a diagram of the complete trellis stage 405 used by the MLSE. 
Each new transition bit shifted into the MLSE represents a new stage 430 in the trellis 
diagram, from 0 to N. At each stage of the trellis, there are 2 L possible states. Each 
possible state in the trellis is referred to as a node 400. Each node 400 has two branches 

15 440, and 420, leading to nodes in the next stage of the trellis. A generalized example of 
the paths between two stages such as 430 and 432 was shown in detail in Fig. 3. In the 
general case, a trellis with symbols from an alphabet of size M will have M branches 
emanating from, and feeding into each node 400. For example, if M=4, there will be four 
branches from each node in a trellis stage, that is one branch to 4 different nodes in the 

20 subsequent trellis stage. Each node in the subsequent trellis stage would have 4 branches, 
one from each of 4 nodes in the previous trellis stage. This pattern would hold true for 
M=8, or any other number. 

The Viterbi Algorithm is implemented in one embodiment of the present 
invention as follows. The path metrics for all states at stage 0 430 of trellis 405 in Fig. 4 

25 are initialized to zero. The branch metrics for each branch 440 leading from each node in 
stage 0 430 to stage 1 432 in trellis 405 are calculated. The manner in which these are 
calculated is shown below. The branch metrics are added to the path metric (zero in this 
case) resulting in a new path metric. For each node (if M=2) in stage 1, there are two 
incoming branches. For example, node 410 may be reached by branch 440, or 450. Each 

30 of these branches are compared to see which has the highest path metric. The higher path 
metric is stored in memory as the state metric. Also, that branch's survivor bit, is stored 
in memory. That is to say, for each node in stage 1 of trellis 405, the path metric of one 
branch (the state metric), and that branch's survivor bit are stored, and one branch's path 
metric and survivor bit are discarded. This process continues from stage 1 to N. At stage 



N, all path metrics are compared, and the one with the highest value is stored. This is 
then the trellis' final state. 

Explained another way, for state q = (qL, qu, qi), the predecessor state 
for predecessor bit 1 is indicated by notation q A l — (qt-i, qL-2? ••■qi? 1) and the predecessor 
5 state for predecessor bit 0 by q A 0 = (qL-i, qL-2, ...qi, 0). For each node of the trellis there 
are two partial paths merging into the node corresponding to predecessor bits 0 and 1 
respectively. The partial paths have associated partial path metrics J n °(q) and J^q which 
indicate the respective likelihood of occurrence. Also, for the two branches merging into 
the node q there is associated an incremental branch metric dJ(q,0) and dJ(q,l) which are 

10 the incremental likelihood respectively of the branches (state transitions) for predecessor 
bits 0 and 1. A partial path metric is equal to the sum of the respective incremental branch 
metric and the state metric of the predecessor state of the branch. The Viterbi Algorithm 
recursively computes the node state metric as equal to the maximum of the two partial 
path metrics merging into the node. This means that the state metric is equal to the larger 

15 of the two partial path metrics of the two paths merging into the node. Among the partial 
paths merging into the node, the predecessor bit of the partial path with the larger path 
metric is termed the survivor bit for the node which is stored for each node. 

At this point, the forward trace is complete, and the backward trace begins. 
Beginning with the final stage N, the survivor bit stored in memory which corresponds to 

20 the highest state metric at stage N is left shifted into a register (as shown in Fig. 2), which 
now hold the final state, as the LSB, thus resulting in a new state at the N-l stage. The 
survivor bit associated with that state in the N-l stage is shifted in as the LSB, resulting in 
the N-2 state. This is then repeated until stage 0 430 is reached. The sequence of MSB 
bits left shifted out of the register is then the received data, corrected for intersymbol 

25 interference. (It may be the reverse of the received data, depending on how the survivor 
bits were stored as they were read out.) 

In other words, at completion of the final stage N, the state q N A with the 
largest metric at stage N represents the terminating state of the maximum likelihood path 
among all trellis paths. The predecessor state along the maximum likelihood path at stage 

30 N-l is next obtained as follows. For terminating state qw A the survivor bit aN(qN A ) is 
retrieved from memory. The shift register of Fig. 2 is initialized with the terminating 
state q N A . Then the shift register contents are shifted left (toward the MSB) and the 
survivor bit is shifted into the LSB position of the shift register. The resultant shift 
register content is the predecessor state qN-i A , and the bit aN A shifted out of the MSB 



position of the shift register on the left is the last symbol of the maximum likelihood 
sequence estimate. Next the survivor bit aN-i(qN-i A ) for state qN-i A is obtained from 
forward trace memory and again shifted into the LSB position of the shift register and the 
shift register content is the next predecessor state qw-2 A on the backward trace along the 
5 maximum likelihood path through the trellis. The bit aN-1 shifted out of the MSB 
position is the next symbol of the maximum likelihood data sequence in reverse order. 
The sequence of survivor bits a>j(qN A ), aN-i(qN-i A ) • • -are thus recursively determined and 
shifted into the LSB position of the shift register and the bits shifted out of the MSB 
position of the shift register represent the maximum likelihood symbol sequence ai A , a2 A , 

10 . . . , aN A in reverse order. 

Fig. 5 shows a simple example of the above for clarity. Here L=2 for 4 
states at each of 4 trellis stages, N=3, and M=2 for two branches into and out of every 
node. Again, the path metrics for each of the four states at stage 0 510 are initialized to 
zero. Branch metrics for each of the 8 branches 501-508 are computed, and added to the 

15 path metrics resulting in a new path metric. At stage 1 520 of the trellis 500, at each 

node, the path metrics are compared, and the higher value metric, along with its survivor 
bit, is stored in memory. (Shown in dark line for clarity.) The lower value path metric is 
discarded. (Shown in dashed line.) For example, state 530 in stage 1 520 of trellis 500 
may have been reached in one of two ways. A zero may have been shifted into 00, path 

20 501. In this case, a 0 is shifted out, and is the survivor bit. Alternately, a 0 may have 

been shifted into 01, path 502. In this case a 1 is shifted out and is the survivor bit. Since 
in this example we chose the path metric for path 501 to be higher, that path metric, along 
with a survivor bit of 0, is stored. This process continues until stage 3 530 of trellis 500 is 
reached. 

25 Fig. 6 shows the branch metrics, referred to here as A-L for simplicity, for 

each of the branches that had the highest path metric of the two branches going into each 
node. The survivor bit for each branch is also shown. When stage 3 610 of trellis 600 is 
reached the path metrics for each state are compared, and the highest identifies the final 
state. For example, at stage 3 610, the path metric for state 00 is J+G+D. For state 01, its 

30 K+F+C, and so on. Stated another way, the path metric at state 00 is J+ the path metric 
up to state 01 at stage 2, which is G+D. 

Assume for example, the highest path metric at the final stage is I+E+C. 
This would mean that 10 is the final state. The back trace is now begun. The survivor bit 
can be seen to be a 0 on branch 620. The zero is shifted into state 10 as an LSB, resulting 



in 00 at stage 2. From state 00 at stage 2, the survivor bit is observed to be a one, leading 
to state 01 at stage 1. The survivor bit here is seen to be a zero, leading us to the 
beginning at state 10 at stage 0. The actual transmitted data must therefore have been 
010, the initial state was 10, and the final state was 10. 
5 The above can be described by the following equations: 



J° (<jr) = J n J (# A 0) + dJ n (q,6) partial path metric, predecessor bit 0 
Jl{q) = J n _ x (q A l) + dJ n (q,i) partial path metric, predecessor bit 1 
dJ n (q,6) is the branch metric for predecessor bit 0 
dJ n (q,l) is the branch metric for predecessor bit 1 
J n (qhma4j°Mjl(q)} 

a n (q) = survivor bit = predecessor bit for partial path with max metric 

10 

This shows that path metrics are computed recursively, that is new branch 
metrics are added to path metrics, resulting in new values for the path metrics. These 
equations define the Viterbi Algorithm as used by one embodiment of the present 
invention. 

15 Reference 2, as cited below, gives the following equations for the branch 

metrics [2]: 



[2] 



dJl fe) = 2^ £ 2„ -s 0 + 2a L s L - 2a L Y,s k a^ 

k=\ 
L-\ 

dJi (q) = 2a L z n -s 0 - 2a L s L - 2a^s k a^ 

k=\ 

q = [q L , q L „ x q x ] trellis state 

1 if q k = 1 

Hk k = 0,l,2,...,L-l,L 

lifq k = 0 



where z n and s k are the MLSE data sources. 



These equations may be simplified by dividing by 2, and adding s 0 . The 
equations will then lose their accuracy as far as the absolute values of the branch matrices 
are concerned. This means they must be reviewed in the context of soft decision 
generation. However, the relative values will remain intact, leading to the correct 
selection of the maximum likelihood path through the trellis. The simplified equations 
are thus [3]: 

dJl = a L z n + a L s L ~ a LYj s k a ^ 

k=\ 

L-\ 

k=i 

[3] 



These equations are then substituted into Equation [1], resulting in the 
following equations [4] which describe the method of the present invention of MLSE 
20 recursive computation of node metric Jn(q) and survivor bit an(q) for state q at stage n of 
the trellis: 
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JO = J n _ x (g A 0)- F([g,0]) + a L z n 
Jl = (? A l)- F([q,l]) + +a L z n 
J n {q) = mzx{jO,Jl} 
a n {q) = binarysign{jO — Jl] 

L-l 

F(l<l,Q]) = ~a L s L +a L Y^ s k a h _^ [4] 

4=1 

^fa4]) = Ffo,0])-2or^ 
? = [?i»?i-i »-,?,] state vector 
_J-1 if? t =0 



In the above equations binarysign {argument} equals 0 if the argument is 
5 zero or positive, and 1 if the argument is negative. These equations are a variant of the 
Viterbi Algorithm that represents the method of the present invention for forward trace 
operation of the MLSE estimation of the transmitted sequence. 

As stated above, the receiver uses the midamble bits 10 in Fig. 1 to create 
the coefficients so, si, . . . sl which model the autocorrelation of the impulse response of 

10 the dispersive channel. Equation 4 defines the branch metrics to be functions of these 
coefficients. If the states of the trellis are sequenced in a Gray code fashion, the 
computation of the branch metrics becomes simplified enough to be performed in real 
time, as required by path metric decision making process. 

Fig. 7 shows an example of this for the 32 states of a system with a 

15 channel memory length L=5. Fig. 7 illustrates by example the use of Gray coding of 
states to substantially simplify branch metric computations according to Equation [4]. 
The example is for the case of channel memory L = 5. There are 2 L = 32 state values q so 
32 pairs of values F(q, 0) and F(q,l) are computed. The Gray coded sequence of states q 
for the example are shown in column 720. Each row of the table shows the state q, F- 

20 Generator calculation of F(q,0) 700, and the computation of F(q,l) 710 For state zero, the 
0 branch metric in column 700 is s, where s= S1+S2+S3+S4+S5. This value may be 
precomputed. To find the 1 branch metric, we simply subtract 2ss as indicated by the first 
entry in column 710. Next, the 0 branch metric for state 1 is computed by subtracting 2s\ 
from the previous entry in column 700, and so on. Similarly for the rest of the first half 

25 of the states, in the n-th row for the new n-th Gray coded state q new , the Gray code bit 
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change occurs in a bit index x. Then F(q,0) is equal to the value of the preceding value of 
F(q,0) in row n-1 for the preceding state q 0 id in the preceding row for preceding state Gray 
coded state q plus or minus 2s x . The minus sign occurs if and only if the Gray code bit 
change is from a zero to a one. For each state value q, the value of F(q,l) is equal to the 
5 value of F(q,0) minus 2s 5 . 

For the second half of the states with most significant bit 1, the state 10000 
is initialized with F(q,0) = -s. From then on, the value of F(q,l) is equal to F(q,0) plus 
2s 5 . Also as before F(q,0) in row n is now equal to the value of the preceding value of 
F(q,0) in row n-1 for the preceding state q 0 id in the preceding row for preceding state Gray 

1 0 coded state q plus or minus 2s x . But now the minus sign occurs if and only if the Gray 
code bit change is from a one to a zero. 

Fig. 7 shows that by using equation 4, and Gray coding, the branch metrics 
may be calculated by simple steps in taken in sequential order. Therefore, the branch 
metrics may be calculated in real time, as they are required. This eliminates the need to 

15 store the branch matrices in memory, as is done in Reference 3, as cited below. This 
means less memory required on the integrated circuit on which the MLSE is placed. 

In order to simplify the branch metric calculations as much as possible, the 
states are not progressed through in pure Gray code. Rather, the first half of the states are 
stepped in Gray code, the MSB is set to one, and the LSBs are stepped in Gray code 

20 again. As an example, in Fig. 7, states 0 to 15 are in Gray code order in the second 

column 720 from 00000 to 01000. At that point, instead of going to 1 1000 for state 16, 
the MSB is set to one, the 4 LSBs are set to zero, and the Gray counting begins again for 
the 4 LSBs from state 10000 to 1 1000. In this way, symmetry between the first and last 
16 states is preserved, and the branch metric calculations simplified. Further, methods 

25 such as taking the complements of previously found values may be used to further 

simplify calculations. In other embodiments of the present invention, the states may be 
arranged in a pure Gray code fashion. Alternately, the state transitions may be limited to 
one bit changing between consecutive states. Alternately, the state transitions may be 
limited to two bits changing between consecutive states. Alternately, the state transitions 

30 may be limited to some other number of bits changing between consecutive states, such 
as 3, or 4. 

In summary, the following may be used to calculate the branch matrices so 
long as the states are Gray coded [5]: 
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F(q new ,Q) = F(q olf ,Q)±2s x 

x = bit index of the state q where bit change occurred in transition q old -» q MW 
± takes negative value when Gray code bit change is from 0 to 1 for msb of q is 0 
± takes negative value when Gray code bit change is from 1 to 0 for msb of q is 1 

± takes negative value if msb of q is 0, positive value otherwise 



The recursive method of branch metric parameter calculation in Equation 
5 [5] according to the present invention requires only one addition per parameter F(q,0) or 
F(q,l). 

In the above described embodiments of the present invention, all the trellis 
initial states were assigned equal likelihood values of zero. This makes sense if no 
information is known about the initial state. However, as an example, in Fig. 1 in a GSM 

10 data burst, the LHS data 20 is preceded by tail bits 25 and dummy bits 22. For L=5, a 

possible known initial state could be 1 1000. By using this initial state, the accuracy of the 
MLSE may be improved. However, as can be seen in Fig. 1, the power is ramping up 
during the transmission of the dummy bits 22. The received bits are corrected based on 
how the channel disperses the midamble, which is sent at the peak power level. Data sent 

15 at different power levels will be affected by the channel differently. Therefore, the first 
1 1 0 of the initial state may be inaccurate. 

One embodiment of the present invention makes use of an initial state 
register, and an initial state mask. The initial state register holds the known initial state of 
length L bits. The initial state mask contains a bit sequence of L bits wherein bit value of 

20 0 in the i-th position indicates that the bit in the i-th position of the initial state register is 
valid. A bit value of 1 in the i-th position indicates a don't care for the i-th position of the 
initial state register. 

As an example, if the initial state register contains 10010, and the initial 
state mask register contains 00000, then the only valid initial state is 10010, since the 

25 mask indicates all the bits in the initial state register are valid. As a further example, if 
the initial state register contains 00000, and the initial state mask contains 0001 1, then 
00000, 00001, 00010, and 00011 are all valid initial states. 

The invalid states likelihood may then be initialized with a very large 
negative value, while valid state are initialized to zero. In this way, the invalid states start 
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at such a large deficit, that path matrices which take their origin from an invalid state 
could never have a higher value than the path matrices which originate from a valid state. 

A further refinement may be made at the end of the trellis. The above 
described embodiments of the present invention selected the highest path matrices at 
5 stage N of the trellis. By adding a final state register, and a final state mask, the selection 
process may be changed to picking the highest path matrices corresponding to a valid 
final state. In this way, the trace back must start and end on valid final and initial states, 
thereby improving the bit error rate performance of the MLSE. 

As an example, in Fig. 8, the initial states 10, and 1 1 are invalid. These 

1 0 states are assigned large negative values -z. The surviving branches are shown, the others 
are deleted. While branches from state 10, and 11 may have higher metrics than each 
other, they can't have a higher metric then paths originating at states 00, and 01. Paths 
starting from invalid states quickly terminate, and if one did make it to the end, it would 
be eliminated by the comparison circuits at that time. In this example, the three valid 

15 final states 00, 01, and 1 1 are compared, and the traceback begins. 

Fig. 9 shows a block diagram of a general linear carrier-modulated digital 
data transmission system 901 that utilizes a MLSE receiver. The major parts of the 
system are the transmitter 905, dispersive channel 915, and receiver 925. The transmitter 
has a coder 900, a modulator 920, and frequency upconverter 940. The coder provides 

20 error correction coding and interleaving of user data to generate a sequence of coded data 
symbols 910. The coded data symbols are digitally modulated onto a carrier by linear 
data modulator 920, and frequency upconverter 940. The resultant signal propagates 
through the dispersive channel 915, which induces intersymbol interference. The signal 
is received by receiver 925, which is optimum in the sense that the receiver determines a 

25 maximum likelihood sequence estimate 955 of the transmitted coded data sequence 910. 
The receiver 925 has a downconverter 970, a low pass filter 990, a ML receiver 945, and 
decoder 965. The received signal is converted in frequency to a complex baseband signal 
980 by the downconverter 970. The baseband signal 980 is filtered by the lowpass filter 
990. The filtered baseband signal 935 is processed by the ML receiver 945 which 

30 generates the MLSE 955 for the transmitter coded data sequence 9 1 0. The MLSE 955 is 
then decoded and deinterleaved by the decoder 965. There are many other possible 
embodiments of this system known to persons skilled in the art. 

Fig. 10 shows a more detailed view of the ML receiver subsystem 945 
which is configured in accordance of the description by Ungerboeck in Reference 2. ML 



receiver 945 has a channel estimator 1020, correlator (not shown), matched filter 1030, 
and MLSE 1080. The received filtered baseband signal 1010 is processed by the channel 
estimator which generates an estimate of the complex channel impulse response h(t) 
1040. The impulse response is time reversed and conjugated to form the matched filter 
5 1030. The filtered baseband signal 1010 is passed through the matched filter 1030 to 

obtain a matched filter output sample sequence zq, zi, 1070 which represent 

observations for the MLSE. The sample period of the matched filter output sequence 
1070 is equal to the coded symbol period of the transmitted digital data 910 in Fig. 9. 
The correlator computes a discrete time autocorrelation sequence of the channel impulse 

10 response so, sj, S2, . . ., sl, where the autocorrelation sample sequences are spaced in time 
at the modulation symbol rate. The parameter L is the length of the channel impulse 
response representing the channel memory. The MLSE 1080 computes the maximum 
likelihood sequence estimate a A (0), a A (l), .... 1090 of the transmitted data sequence based 
on processing of observations z 0 , z u 1070 and channel autocorrelation parameters s 0 , 

15 si, S2, . . Sl 1050. In summary, the MLSE outputs maximum likelihood estimate a A (0), 

a A (l), .... 1090 based on input data sources zo, zi, 35 1070 and so, Si, S2, . . ., s 1050. It is 

understood by persons skilled in the art that although the original theory of maximum 
likelihood receivers as described for example in References 1 and 2, cited below, apply to 
linear modulations, the theory can be usefully extended to certain nonlinear modulations 

20 as well such as GMSK by means of suitable linear approximations. 

The computed maximum likelihood sequence estimates 1090 that are 
output by the MLSE are inherently hard decisions; that is, the decision a n A for the n-th 
symbol has a value from the discrete symbol alphabet of the modulation data 910. It is 
well known that soft decision decoders that utilize available soft decision metrics 

25 generally have superior error rate performance relative to decoders that utilize only hard 
decision information. The MLSE can be optionally augmented with a soft decision 
generator 1055 which provides a soft decision metric b n 1060 for each symbol hard 
decision a n A output by the MLSE. The soft metric is a measure of the certainty of the 
hard decision. The MLSE hard decisions 1090, and optionally the corresponding soft 

30 metrics 1060, are output to the decoder 965. 

A computational method typically used for the of the MLSE 1080 is 
known in the art as the Viterbi algorithm and one embodiment of the present invention 
uses a variant of the Viterbi algorithm which can be summarized as follows. The MLSE 
operates on the observation sequence zq, z u 1070 which are sufficient statistics, that is, 
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contains all relevant information for maximum likelihood estimation of the transmitted 
sequence 910. For any hypothesized transmitted sequence ao, ai, ... there is an associated 
state sequence representation ... o"o, oi, .... where cr n = [a n , a n _i,...,a n -L], and L is the channel 
memory. Determination of the maximum likelihood symbol sequence is equivalent to 
5 determination of the corresponding maximum likelihood state sequence so that the two 
are mentioned interchangeably. 

Fig. 1 1 shows the preferred embodiment of the MLSE 1080 according to 
an embodiment of the present invention. The MLSE comprises an Arithmetic Logic Unit 
(ALU) 1 130, F-Generator 1 120, FSM 1 125, and Memory 1 105. The ALU 1 130 provides 

10 computational means for both forward trace of the Viterbi Algorithm computation in 
accordance with Equations [4] and backward trace computation. The Memory 1 105 
provides data storage means in support of the ALU computations. The Memory 1 105 
provides means for storage of four types of data which are stored in respective memory 
modules 1135, 1136, 1137, and 1138. The memory module 1137 labeled State Memory 

15 Mem J has address variable addrj (not shown) which provides means for reading and 

writing path metric values J n (q) by the ALU 1 130 for all 2 L states q and for trellis stages n 
from 1 to N. Mem J need only hold two stages of metric values at any time, current 
stage n and past stage n-1, since the data storage locations for the two stages are 
alternated in Ping-Pong fashion. That is, the ALU 1130 overwrites data storage locations 

20 of Mem_j holding stage n-2 node metrics with node metrics of current stage n because the 
stage n-2 node metrics are no longer needed in the recursive computations of Equations 
[4] for J n (q). The ALU 1130 reads data from the specific address of MemJ holding 
predecessor state metrics (n-1) when needed in the ALU 1 130 computations. During 
forward trace, the memory module 1138 labeled Trace Memory Mem_a having address 

25 variable addr_a (not shown) provides means for writing and reading of survivor bits a n (q) 
for stage n and state q by the ALU for each of the Nx2 L nodes q of the trellis. The data 
a n (q) 1 165 is written into Mem_a 1 138 by the ALU 1 130 during the forward trace 
operation of the MLSE 1080. During backward trace the ALU reads the sequence of 
survivor bits a n (q n A ) 1 160 for n = N, N-1, .... from Mem_a 1 138 to determine the 

30 corresponding maximum likelihood symbol a n A which is rewritten back into Mem_a 

1138. The memory module 1 136 labeled Observation Memory Mem_obs with address 
variable addr_obs (not shown) provides means for writing and reading of the message 
output sequence Zn, n = 1,2, N 1 140 of the matched filter 1030 which serves as a data 
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source for the MLSE 1080. The ALU reads the data from Mem_obs 1 136 as needed to 
compute the path metrics accordance with Equations [4]. The memory module 1135 
labeled Correlation Memory Mem_cor provides means for reading and writing of the L+l 
data elements corresponding to twice autocorellation values 2si, 2s 2 , 2s L , 1 1 10 of the 
5 channel impulse response and the sum of autocorrelation values s = s 0 + Si + .... + s L . The 
data elements stored in Mem_cor 1 135 are computed and written to Mem_cor 1 135 by 
the correlator, and made available to the F-Generator 1 120 as needed to compute branch 
metric parameters F(q,0) and F(q,l) in accordance with Equations [4]. 

For the optional Soft Decision Generator 1055 of Fig. 10, the ALU 1030 
10 outputs partial path metrics JO and Jl for each node to support the soft decision 
computations. 

The Finite State Machine (FSM) 1125 provides means for control and 
sequencing of the MLSE operation via control signals 1135. Thus, the FSM control 
signals include designation of the addresses addrj, addr_a, and addr_obs for reading and 

15 writing of specific data by the ALU from respective memory modules 1137, 1138, and 
1136. The controls also include sequencing of the current state q, and the current stage n, 
where the pair (n,q) designate a specific node in the trellis during forward trace operation 
through the trellis. The trellis stage sequencing is n = 1,2, N, and for value of n, the 
state values q sequence through a Gray code sequence as described above, and again here. 

20 The ALU 1130 computes and F-Generator 1 120 jointly compute the state metric J n (q) and 
survivor bit a n (q) for each node (n,q) in accordance Equations [4]. The FSM provides 
means for control of the operation of signal processing elements such as latches, 
multiplexers, adders, and registers comprising the ALU and F-Generator. The MLSE 
operation under FSM control operates in four phases: (1) forward trace initialization, (2) 

25 forward trace, (3) backward trace initialization, and (4) backward trace. The phases are 
sequential except that backward trace initialization phase overlaps with the final stage N 
of the forward trace phase. 

The F-generator 1 120 provides means for computation of branch metric 
parameters F(q,0) and F(q,l) for the current state q in accordance with Equations [4], 

30 where the computations are in real time, that is, computed when needed by the ALU 

1130. For each stage of the trellis, the sequencing of the 2 L states q by the FSM 1 125 is 
according to a Gray code. In Fig. 7, an example of a Gray coded state sequence was 
shown for the 32 states for the case of channel memory length L = 5. The Gray coding 
of the state sequence substantially simplifies the computation of the branch metric 
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parameters F(q,0) and F(q,l) in the F-Generator. The method initializes and performs the 
calculation for the first half of the states with the most significant bit 0, and then performs 
the calculation for the second half of the states with most significant bit 1 . 

Fig. 12 shows the an embodiment for the F-generator 1 120 consistent with 
5 the present invention and suited for VLSI implementation. The F-Generator 1 120 

provides means for executing the novel recursive method of computation of Equations 
[5]. The F-Generator comprises signal processing components including latches 1220 
and 1210 labeled F and Flast respectively, adder 1260, multiplexers 1230 and 1240 
labeled Mux 1 and Mux 2 respectively, and sign inverter 1250. Multiplexer 1240 has 

10 L+l selectable input sources, where the input sources are respectively the L+l channel 
autocorrelation data words stored in memory Mem_cor 1135 of Fig. 1 1 . The sign inverter 
1250 is a single-input single output device which provides means for passing data from 
input to output either with or without data inversion. The operation of the signal 
processing components are controlled by the FSM control signal with supporting signal 

1 5 processing clock (not shown) . Latch F 1220 alternately presents the values of the branch 
metrics F(q,0) and F(q,l) to the ALU 1130 of Fig. 1 1 for the Gray coded sequence of 
state values q according to Equations [5] by means of the following signal processing 
steps at each stage n of the trellis. 

Step 1 . The initial state q is set to the initial all-zeros state. 

20 Step 2 . Multiplexer 1230 selects as input the register (not shown) holding 

value zero and the multiplexer Mux 2 1240 selects as input the Memcor memory 1 135 
location for parameter s. The inverter 1250 is set for no inversion. The latch F 1220 
receives value s which is the value F(q,0) for the initial all-zeros state. The F latch 
content F(q,0) is an output 1170 available to the ALU 1130 to support computation of 

25 Equation [4] for initial state q. 

Step 3 . Latch Flast 1210 receives the value in latch F 1220 so that Flast = 

F(q,0). 

Step 4 . Multiplexer Mux 1 1230 selects latch Flast 1210 as input. 
Multiplexer Mux 2 1240 selects as input the Memcor memory 1 135 location storing 
30 parameter 2s L (alternately, the parameter sl may be stored, and left shifted one bit to 
multiply it by 2). The inverter 1250 is set so that its output value is the negative of its 
input value. The input to adder 1260 is then the output of latch Flast 1210, and negative 
2s L . The input to latch F 1220 is then equal to Flast - 2s L = F(q,0) - 2s L = F(q,l). The 
latch F value F(q,l) for state q is then presented as an output 1220 to the ALU 1 130. 



Step 5 . The FSM increments the Gray coded state q, that is, the next Gray 
coded state q new is set by the FSM resulting in a bit change relative to the preceding old 
state vector q 0 ]d at bit index x of the state. That is q n is incremented in a Gray code 
fashion to the next state in the second column 720 in Fig 7 
5 Step 6 . Multiplexer Mux 2 1240 selects as input the Mem_cor memory 

1 135 location for parameter 2s x . The inverter 1250 is set for sign inversion if and only if 
the Gray code state change in Step 5 is a bit change from a 0 to a 1 . Then latch F = Flast 
+/- 2s x where the negative sign applies if the Gray code bit change is from a 0 to a 1. 
Since Flast is the value F(q oW ,0) for preceding state q 0 id, then F = F(q o id,0) +/- 2s x = 
10 F(q new ,0). The value of F(qnew,0) held in latch F 1220 is presented as an output 1 170 to 
the ALU 1130. 

Step sequence 5.6.3.4 ar e repeatedly executed by the F-Generator for 
each state value through the final Gray coded state value. 

There are obvious variations on the preferred embodiment for computing 

15 the pair of branch metric parameters F(q,0) and F(q,l) according to Equations [5]. The 
value of F(q,0) for state q is inverted if the state q is complemented. That is, F(q,0) = - 
F(q*,0) where q* is the complement of q. Therefore, in one variation, only half the 
parameter values are computed as in the preferred embodiment, and the other half are 
obtained by inversion. For example in Fig. 5, the Gray coded states with MSB = 0 can be 

20 calculated as in the preferred embodiment, and for each state q thus calculated, F(q*,0) 
and F(q*, 1) for the complemented state can be obtained simply by inverting the values of 
F(q,0) and F(q,l) respectively. The set of all complemented states correspond the 
remaining half of the states with MSB = 1 . 

Fig. 13 shows a flow chart for the branch metric calculations consistent 

25 with one embodiment of the present invention. At act 1300 the state is initialized to zero. 
Parameter s is read from memory as F(q,0) in act 1305, and output in act 1310. Branch 
metric F(q,l) is then computed from F(q,0). If the MSB of the present state is zero, then 
2sl, where L is the channel memory, is subtracted from F(q,0) resulting in F(q,l) in act 
1320. If the MSB of the present state is one, then 2s L , where L is the channel memory, is 

30 added to F(q,0) resulting in F(q, 1) in act 1 325 . 

As discussed above, in order to simplify the branch metric calculations as 
much as possible, the states are not progressed through in pure Gray code. Rather, the 
first half of the state are stepped in Gray code, the MSB is set to one, and the LSBs are 
stepped in Gray code again. As an example, in Fig. 7, states 0 to 15 are in Gray code 



order in the second column 720 from 00000 to 01000. At that point, instead of going to 
1 1000 for state 16, the MSB is set to one, the 4 LSBs are set to zero, and the Gray 
counting begins again for the 4 LSBs from state 10000 to 11000. The net result of this is 
that codes 16 to 3 1 are stepped through in reverse order. In this way, symmetry between 
5 the first and last 16 states is preserved, and the branch metric calculations simplified. 

Further, methods such as taking the complements of previously found values may be used 
to further simplify calculations. Another embodiment of the present invention uses Gray 
coding in a pure Gray code sequence for all states. What is preferred is that only bit 
change between consecutive states. Other embodiments are possible. For example, a 

10 sequence of arranging the states where either one or two bits are changed between 

consecutive states could be used. In that case two computations per state change may be 
required. Alternately, a sequence of states where three or less, or some number greater 
than three, such as four, or less, bits are changed between consecutive states could be 
used. Accordingly, a greater number of computations may be required for each state 

1 5 change. 

As described above, for one embodiment of the present invention, the 
counter counts Gray code from 0 to 15 with the MSB = 0. Then the MSB is set to 1, and 
the four LSBs repeat counting from 0 to 15 again. Therefore, acts 1330 and 1335 check 
to see if 01000 or 11000 have been reached. If 01000 has been reached 1330, the state is 

20 set to 10000 act 1380, -s is read in as F(q,0) act 1375, and output in act 1310. If 1 1000 
has been reached, the branch metrics for the last state have been computed, and the 
process comes to a stop in act 1370. If 01000 or 1 1000 have not been reached, the state is 
incremented in Gray code fashion acts 1340, or 1345 as appropriate. Next the bit that was 
changed is observed as to whether it was a 0 to 1 transition. If the answer is yes and the 

25 MSB is zero, or the answer is no and the MSB is one, then 2s x is subtracted from the 
F(q,0) of the previous state to generate the F(q,0) for the present state in act 1360. X is 
the bit position that was changed between the previous state and the present state. If the 
answer is no and the MSB is zero, or the answer is yes and the MSB is one, then 2s x is 
added to the F(q,0) of the previous state to generate the F(q,0) for the present state in act 

30 1365. F(q,0) is then output in act 1310, and the process repeats. In this example, L was 
chosen to be 5, but the example is applicable for L of any value with simple changes such 
as counting to 0100 and 1100, or 010000 and 110000 if L=4 and 6 respectively. 

Fig. 14 illustrates the preferred embodiment for the forward trace 
computation of the state metrics J n (q) and survivor bits a n (q) by the ALU 1 130 in 



accordance with Equations [4] as used by one embodiment of the present invention. The 
ALU signal processing for forward trace operation comprises six multiplexers 1410, 
1415, 1420, 1425, 1430, and 1435 labeled Mux A, Mux B, Mux C, Mux D, Mux E, and 
Mux F respectively, five latches 1440, 1445, 1450, 1455, 1460, 1465, and 1470, labeled 
5 JO', Jl', z, Diff, JO, Jl, and AJ respectively, one adder 1472, two subtractors 1475 and 
1477, a sign inverter 1479, and a sign operator 1481. The subtracter 1475 computes as 
output the Mux D 1425 latched value minus the Mux C 1420 latched value. The 
subtractor means 1477 computes as output the JO latch value minus the Jl latched value. 
The output of sign inverter means 1479 is equal to its input if the MSB of state q is 1, and 

10 otherwise the output is the negative of the input. The sign operator means 1481 outputs 
the sign bit of the latch AJ 1470 value expressed as a twos complement number; that is, 
the output is 1 if the latched value in A J 1470 is negative and the output is 0 if the latched 
value in AJ 1470 is zero or positive. 

The ALU described in Fig. 14 provides for the following sequence of 

15 signal processing steps under control of the FSM. In the first phase, the FSM resets State 
Metric Memory MemJ so that all addresses corresponding to the initial stage metrics 
Jo(q) for all 2 L states q contain zero value. In the second phase, the FSM sequences the 
state or node designations of the trellis (n,q), from stage to stage n = 1, 2, N, and for 
each stage sequences the states q through all 2 L Gray coded states, or modified Gray code 

20 states as described above. For each node or state (n,q) of the trellis, the ALU computes 
the survivor bit a„(q) and state metric J n (q) in accordance with Equations [4] in the 
following way. 

Step A . Address addrj is set for reading of memory Mem J 1 137 the 
storage location for the state metric J n .i(q A 0) 1 150 for predecessor state q A 0 
25 corresponding to predecessor bit 0. 

Step B . Mux A 1410 selects input from memory MemJ 1 137. Latch JO' 
1440 receives stored value in MemJ(addrJ). That is JO' = J n _i(q A 0). 

Step C . Address addrj is set for reading of memory Mem J 1 137 the 
storage location for the state metric J n .i(q A l) for predecessor state q A 2 corresponding to 
30 predecessor bit 1. 

Step D . Mux C 1420 selects as input the F-Generator output 1 170, Mux D 
1425 selects JO' 1440 as an input, and Mux A 1410 selects Diff as an input. Then, JO' = 
Diff = JO' - F = J n .i(q A 0) - F. As presently described the F-Generator control will be 
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phased such that the F-Generator output F - F(q,0) at this step so that JO' - J n -i(q A 0) - 
F(q,0). 

Mux E 1430 selects input from memory MemJ 1137. Latch Jl' 1445 
receives stored value in MemJ(addrJ). That is Jl' = J n -i(q A l)- 

Step E . Mux C 1420 selects as input the F-Generator output 1 170, Mux D 
1425 selects the output of Jl ' 1445 as an input, and Mux B 141 5 selects Diff 1405 as an 
input. Then, Jl ' = Diff = Jl '- F = J n -i(q A l) - F. As presently described the F-Generator 
control will be phased such that the F-Generator output F = F(q, 1) at this step so that Jl ' 
= J n . 1 (q-l)-F(q,l). 

Step F . Mux E 1430 selects the output of JO' 1440. The execution of this 
step under FSM control coincides with Step D so that JO' = J n .i(q A 0) - F(q,0). The 
address addr_z of memory Mem_z 1140 is set to the location of observation variable z^ 
for reading by the ALU 1 130. Latch JO 1440 is enabled so that it receives partial path 
metric JO = JO' +/- z n . = J n -i(q A 0) - F(q,0) +/- z^. 

Step G . Mux E 1430 selects the output of Jl ' 1445. The execution of this 
step under FSM control coincides with Step E so that Jl ' = J n -i(q A l) - F(q,l). The 
address addr_z of memory Mem_z is set to the location of observation variable Zn for 
reading by the ALU. Latch Jl 1465 is enabled so that it receives partial path metric Jl = 

Jl' +/- ^ = J„-i(q A l) - F(q,l) +/" zn- 

Step H. The address addr_a 1 160 of Trace Memory Mem_a 1 138 is set for 
writing to the storage location for survivor bit a n (q). The address addrj 1 145 of State 
Metric Memory MemJ 1 137 is set for writing to the storage location for state metric 
Jn(q), 

Step I . Latch AJ 1470 is enabled so that A J = Jl - JO gets the differential 
of the path metrics. The output 1490 of the sign operator 1481 is the survivor bit a n (q), 
i.e., binarysign{J0-n} which has binary value 0 if JO is greater than or equal to Jl and 
binary value 1 if Jl is greater than JO. The output a n (q) 1481 is written to Mem_a 1 138. 

Step J . Mux F 1435 selects the output of JO 1460 if a n (q) is 0 or the output 
of Jl 1465 if a n (q) is 1. The Mux F 1435 output 1495 J n (q) = max{J0,Jl} is written to 
memory MemJ 1138. 

The preceding signal processing steps describe the method of the preferred 
embodiment of the MLSE Viterbi Algorithm for forward trace operation according to 
Equation [4]. The processing can be implemented in VLSI hardware wherein multiple 
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steps are performed in parallel under FSM control and data processing clock. Table 1 
summarizes the parallel processing steps and phasing of the processing clock for the ALU 
and F-Generator. Each state q at each trellis stage n requires computation of survivor bit 
a n (q), state metric J n (q), and supporting branch metric parameters F(q,0) and F(q,l). After 
initialization, the signal processing steps of the ALU and F-Generator are both performed 
in parallel in two clock cycles per state. The first row of Table 1 indicates the processing 
clock phasing or modulus value 0 or 1 . The second row indicates at which clock phase 
each ALU processing step is performed. The third row indicates at which clock phase 
each F-Generator processing step is performed. In Table 1, the relative timing of the 
processing provides at F-Generator output 1 170 the values of computed branch metric 
parameter values F(q,0) and F(q,l) precisely when needed by the ALU processing. In 
this way, the branch metrics are computed in real time, and are available to the ALU 1130 
when required to compute the path metrics. The ability to compute the branch metrics in 
real time eliminates the need to precompute them and store them in memory, thus 
reducing the die size of an integrated circuit implementing a MLSE in accordance with 
the present invention. 

Steps 1 through 7 for the F-Generator 1 120 and Steps A through G for the 
ALU 1130 describe the signal processing steps associated with forward trace operation of 
the MLSE Viterbi Algorithm for the preferred embodiment in accordance with Equation 
[4]. The processing can be implemented in VLSI hardware and multiple steps are 
performed in parallel under FSM control and associated data processing clock. The signal 
processing steps of the state metrics and survivor bits for each node of the trellis are 
computed in parallel with two clock cycles per state metric calculation under FSM 
control. The parallel operations are Steps 3, 4, A, D, and F executed on odd clock cycles 
and Steps 1, 2, 5, 6, B, C, and E executed on even clock cycles. Since the F-Generator 
and ALU 1130 perform in parallel, the branch metrics may be computed "on the fly" in 
real time as needed, thus eliminating the requirements of precomputing and storing the 
branch metrics ahead of time. This means not only as discussed above, that the integrated 
circuit memory may be reduced, but that such reduction does not occur at the cost of extra 
clock cycles. 

Following forward trace operation, the third FSM phase is backward trace 
initialization, followed by the fourth FSM phase of backward trace operation. Fig. 15 
shows the preferred embodiment of the ALU 1130 signal processing that supports 
backward trace phase and the preceding initialization phase. The ALU 1 130 for the 
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backward trace operation comprises a metric comparator 1500, and a shift register 1510. 
The Metric Comparator 1500 provides means for calculation of the final trellis state q N A 
on the maximum likelihood path through the trellis. The final trellis state q N A has 
maximum metric among all states q in the final trellis stage N, and serves as the initial 
state in the backward trace phase of operation. The Metric comparator is enabled by the 
FSM only during the Viterbi Algorithm calculations for the final stage N of the trellis. 
The inputs to the Metric Comparator are the sequence of states q 1 1 3 5 and the 
corresponding calculated state metrics J N (q) 1490 for trellis stage N, and at the end of 
stage N, the output 1501 from the Metric Comparator is the final state q N A of the 
maximum likelihood path. 

The backward trace initialization phase is provided by the following ALU 
apparatus. The Metric Comparator 1500 comprises two multiplexers 1520 and 1522 
respectively labeled Mux G and Mux H, two latches 1524 and 1526 respectively labeled 
q_max and J_max, a subtractor 1528, and a sign operator 1529. The subtractor 1528 
computes the value of the new state metric J N (q) minus the value of the state metric in 
latch J_max in two's complement arithmetic and the 2-s complement sign of the 
remainder is output by the sign operator 1529 as the control signal 1537 for the two 
multiplexers. When the control signal has value 0, the new state metric is larger than the 
contents of latch J_max, Mux H 1522 selects the input as the new state metric value J n (q), 
and Mux G 1520 selects the new state value q on line 1 135. The signal on control line 
1537 thereby causes the latches J_max 1526 and q_max 1534 to hold the largest state 
metric and associated state as the ALU sequentially computes the state metrics J n (q) for 
final stage N of the trellis. At the end of forward trace computation at stage N, the 
contents q N A in latch q_max 1524 is the state having the largest final state metric. The 
ALU 1 130 provides means for shifting the q N A into the shift register 1510 which 
completes initialization of the backward trace operation. 

In Fig. 15 the shift register 1510 is the realization of the conceptual shift 
register of Figure 2. Thus, the shift register 1510 provides for storing a state vector of L 
bits, where L is the dispersive channel memory length in symbol lengths. The shift 
register means provides for shifting of the state vector one symbol toward the MSB 
direction (left), with shift register input 1513 of one bit into the vacated LSB bit position, 
and shift register output 1563 of one bit corresponding to the bit that vacated the MSB bit 
position of the shift register. The FSM now controls the trace back operation by 
sequentially down counting the trellis stage index n so that n = N, N-l , .... , 1 . For the 
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traceback routine at trellis stage N, the initial shift register state q N A loaded during 
backward trace initialization phase is sent to the Trace Memory Mem_a 1138 and the 
survivor bit a N (qN A ) is read out of Mem_a 1138 from readout address addr_a = [N, qN A ]. 
The shift register shifts one bit left and the survivor bit a N (q N A ) is input to the shift 
5 register at the LSB position and the MLSE estimate a N A for the final symbol is output 
from the MSB position of the shift register. The ALU 1130 provides means to write shift 
register MSB output a N A 63 into Mem_a 1138 at address addr_a = [N, 0]. The shift 
register now contains the state qN-i A on the maximum likelihood path at stage N-l . The 
traceback routine for stage N is repeated for stage N-l . That is, shift register state qN-i A is 

10 used to read the survivor bit a N .i(q N -i A ) from memory Mem_a 1 138 and the survivor bit is 
then input to the vacated shift register LSB position when the shift register state vector is 
shifted one bit left. The output of the shift register from the MSB position a N _i A 
corresponds to the MLSE estimate for symbol index N-l and is stored in memory Mem_a 
at address addr_a = [N-l, 0]. The traceback routine is thus repeated sequentially for 

15 stages n = N-2, N-3, 1. Then at termination of the traceback routine for stage n = 1, 
the MLSE estimate ai A , a 2 A , a N A of the transmitted data sequence 910 is contained in 
memory Mem_a 1138. If the address addr_a = [n,q] is interpreted as the n-th row and q- 
th column of an Nx2 L memory array for Mem_a, then the MLSE estimate is stored in the 
first column of Mem_a ready for usage by the decoder 965. 

20 Table 2 summarizes the number of processing clock cycles required for the 

preferred embodiment of the present invention to generate a maximum likelihood 
sequence estimate of a block of N transmitted data symbols 910. The required 
processing clock cycles is shown for each of the four phases of MLSE operation under 
FSM control: (1) initialization of the state metric memory for forward trace, (2) forward 

25 trace calculations for state metrics J n (q) and survivor bits a n (q), (3) backward trace 

initialization, and (4) backward trace to obtain for maximum likelihood sequence estimate 
a n A from Trace Memory. Table 2 shows that the total number of clock cycles required to 
process a block of N symbols is (2 L x2xN)+N cycles. 

As a numerical example, the GSM standard for cellular 

30 telecommunications [Ref. 4] is considered. For the GSM standard, data is transmitted in 
bursts within time-division slots, and Fig. 1 illustrated the burst data format for a slot. A 
natural processing block for the MLSE is the block of N = 58 data bits wherein the 
Viterbi Algorithm is computed first for the LHS data block, and then independently for 
the RHS data block. For GSM the maximum channel memory is L = 5 symbol lengths. 



Then evaluating Table 2 for the case L = 5 and N = 58, the preferred embodiment of the 
MLSE of the present invention requires 3778 cycles per block and 7556 cycles for the 
two blocks of the data burst. A GSM slot is 577 microseconds in length and the MLSE 
can complete the 7556 clock cycles of processing in 378 microseconds with a 20 MHz 

5 processing clock or in less than a slot duration. Then for reception of back-to-back slots 
in a so-called multi-slot mode provided for in the GSM specifications, the MLSE of the 
present invention is completes the processing of a slot of data with substantial margin 
before the next slot of data arrives 

Table 3 summarizes the memory requirement for the preferred 

10 embodiment of the present invention. Column three shows that for the GSM example 
where N = 58 bits and L = 5, the total memory requirement is 2720 bits. This assumes 
typical word lengths for GSM of 16 bits for correlation parameters stored in memory 
Mem_cor 1135 and for branch metric parameters F(q,0) and F(q,l), and 22 bit word 
lengths for state metrics stored in memory Mem J 1 137. 

15 As discussed above in Fig. 8, the MLSE performance can be improved, 

that is Bit Error Rate (BER) can be reduced if the possible paths through the trellis are 
limited to those beginning and ending at known valid states. The method of the invention 
for utilizing initial and final state information in an alternative embodiment to enhance 
Viterbi Algorithm performance is shown with the aid of Figures 16 and 17. The MLSE is 

20 the same as the preferred embodiment except for the modifications described in Figs. 16 
and 17. Fig. 16 shows modifications to the ALU that incorporate known information 
about the initial state into the Viterbi Algorithm during state metric initialization. 
Relative the previous embodiment this embodiment additionally includes in the ALU 
1130 an Initial State Register 1640, an Initial State Mask 1642, a comparator 1644 which 

25 tests for equality of Initial State 1640 and current state q 1 135 subject to the Initial State 
Mask 1642. This embodiment further includes lines to accept two new inputs for the 
multiplexer 1435 labeled Mux F from the preferred embodiment of Figure 14. The two 
new inputs are a 24-bit constant -C 1645 and 24-bit value zero 1643. The Initial State 
register 1640 holds the known initial state of length L bits. As discussed above, the 

30 Initial-State Mask register 1 642 contains a bit sequence of length L bits wherein a bit 

value 0 in the i-th bit position indicates that the symbol in the i-th bit position of the initial 
state register 1 640 is a valid bit. A bit value 1 in the i-th bit position indicates an invalid 
or don't-care logic for the i-th bit position of the initial state register 1640. The 
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combination of the Initial State and the Initial State Mask jointly define all valid initial 
states. 

The configuration of Fig. 16 for the alternative embodiment applies only 
during stage n = 0 of the trellis, corresponding to a state metric initialization phase, and 
5 the ALU operation is otherwise the same as the preferred embodiment during the forward 
trace phase of the Viterbi Algorithm. The FSM cycles through all 2 L state values q 1135 
during trellis stage n = 0. The value of the comparator output 1641, which is also the 
Mux F 1435 control signal is binary 0 if the current state value q is a valid initial state and 
is binary 1 otherwise. A control value 0 causes Mux F 1435 to select as input the value 

10 zero 1643 and a control value 1 causes Mux F 1435 to select as input the value -C 1645 
which is a negative number with large magnitude. The Mux F 1435 output 1495 
represents initial state metrics J n (q) = Jo(q) and is stored to state metric memory MemJ at 
address addrj specified by the FSM. Then, the valid initial states receive relatively high 
initial state metric values (zero) compared to invalid states which receive negative initial 

15 state metric values -C. The constant C has sufficiently large magnitude that the 

maximum likelihood path finally selected by the Viterbi Algorithm will always start on a 
valid state. The metric word size in the preferred embodiment for the GSM application is 
24 bits and the branch metric word size is 16 bits. The constant C has value 2 16 . After 
the FSM cycles the ALU 1130 through the preceding initialization phase using the 

20 modifications of Figure 16 to the preferred embodiment, the remaining ALU forward 

trace processing phase is identical to the preferred embodiment of Figure 14. In Table 2, 
the initialization phase replaces the reset, and requires 2 L clock cycles to initialize the 2 L 
state metrics at stage n = 0 of the trellis. 

Fig. 17 shows modifications to the metric comparator 1500 of Figure 15 

25 for an alternative embodiment that incorporates known information about the final state 
into the Viterbi Algorithm during final stage N of the trace forward operation. It is to be 
recalled that the metric comparator 1500 determines the final state q N A of the maximum 
likelihood path which serves as the initial state for Viterbi Algorithm backward trace 
operation. The alternative embodiment additionally includes in the metric comparator 

30 1500 a Final State register 1730, a Final State Mask register 1732, a comparator 1734 
which tests for equality of Final State 1730 and current state q 1135 subject to the Final 
State Mask 1522. The output 1537 of the comparator 1534 has binary value 0 if the 
masked equality holds, that is the current state q 1 135 is a valid state. The alternative 
embodiment further includes a logical-and operation 1736 which has an output that 



controls the multiplexers 1520 and 1522 and has value 0 if and only if the sign operator 
1529 output and comparator 1737 output both have zero value. The latch q_max 1524 
latches the current state only if the state is a valid final state and its state metric is larger 
than the currently latched state metric value. As a consequence, at the end of final trellis 
5 stage N the latched value q N A in latch q_max is the state having the largest state metric 
among the set of all valid terminal states. In this way the alternative embodiment forces 
the Viterbi Algorithm to confine search for the maximum likelihood path to the set of all 
paths having a valid terminal state as well as a valid initial state, thereby enhancing the 
performance of the Viterbi Algorithm. Thereafter the backward trace operation is 

1 0 identical with that of the preferred embodiment. 

As an example of usage prior initial or final state information in this 
embodiment, Table 4 summarizes the register contents indicating prior knowledge about 
initial and final states for both LHS data block and RHS data block of a data burst for the 
GSM example. In this example, channel memory L = 5 and trellis length N = 63 (58 data 

1 5 symbols + 5 symbols extension to the known final state). 

In the one embodiment the MLSE required only two cycles of the 
processing clock per state in the forward trace operation of the Viterbi Algorithm. 
However, the State Metric memory is required to support simultaneous read/write access 
which implies increased memory complexity such as dual port memory. This is because 

20 for each state as in the preferred embodiment there are two memory accesses for reading 
of predecessor state metrics, and one writing of current state metrics. In an another 
embodiment the MLSE requires three clock cycles but has the benefit of less complexity 
in the ALU 1130 and memory. An alternative ALU 1 130 is shown in Fig. 18. In addition 
to the lesser number of components compared to the embodiment in Fig. 14, this 

25 embodiment does not require simultaneous read/write memory access to State Memory 
Mem J 1137. 

The ALU 1 130 comprises data processing components including six 
multiplexers 1862, 1864, 1866, 1868, 1870, and 1871 labeled Mux A, Mux B, Mux C, 
Mux D, Mux E, and Mux F respectively, five latches 1872, 1873, 1874, 1875, 1877, 
30 labeled JO, Jl , z, Diff, and Memreg respectively, two adders 1 878 and 1 879, a threshold 
tester 1876, and a sign inverter 1867. The adder 1878 computes as output the Mux D 
1868 output minus the Mux C 1866 output. The adder means 1879 computes the sum of 
Mux E 1870 output and the output of sign inverter means 1867. The output of sign 
inverter means is set equal to the input if the MSB of state q is 1, and otherwise the output 



is the negative of the input. The threshold tester 1876 tests whether Diff is <0 and outputs 
a binary 0 if true and binary 1 otherwise. 

In Fig. 18, the ALU 1130 provides for forward trace computation of the 
Viterbi algorithm in accordance with Equations 4 through the following signal processing 
5 steps. The FSM cycles the ALU 1130 from stage to stage n = 1, 2, N of the trellis, and 
for each stage cycles through all states q in a Gray coded sequence, or a modified Gray 
code sequence as described above. In Fig. 18, at each state q of stage n, the ALU 1 130 
provides means for computation of the survivor bit a n (q) according to Equations [4] under 
control of the FSM as follows: 
10 Step A . Address addrj is set for reading of memory Mem J 1 137 the 

storage location for the state metric J„-i(q A 0) for predecessor state q A 0 corresponding to 
predecessor bit 0. 

Step B . Mux A 1862 selects input from memory Mem J 1137. Latch JO 
1872 receives stored value in MemJ(addrJ). That is JO = J n _i(q A 0). 
15 Step C . Address addrj is set for reading of memory Mem J 1 137 the 

storage location for the state metric J n -i(q A l) for predecessor state q A 2 corresponding to 
predecessor bit 1. 

Step D . Mux C 1866 selects as input the F-Generator output 1 170, Mux D 
1868 selects the output of JO 1872 as an input, and Mux A 1862 selects Diff as an input. 
20 Then, JO = Diff = JO- F = J n -i(q A 0) - F. As presently described the F-Generator control 
will be phased such that the F-Generator output F = F(q,0) at this step so that JO = J n - 
Kq'XQ-FfoO). 

Mux E selects input from memory Mem J 1137. Latch Jl 1873 receives 
stored value in Mem J (addrj). That is Jl = J n .i(q A l). 
25 Step E . Mux C 1 866 selects as input the F-Generator output 1 170, Mux D 

1868 selects the output of Jl 1873, and Mux B 1864 selects Diff as an input. Then, Jl = 
Diff = Jl- F = J n _i(q A l) - F. As presently described the F-Generator control will be 
phased such that the F-Generator output 1170 F = F(q,l) at this step so that Jl = J n -i(q A l) 
-F(q,l). 

30 Step F . Address addr_a is set for writing to trace memory Mem_a 1 138 at 

the location for node (n,q) for stage n, state q. JO and Jl do not update. Mux C 1866 
selects the output of JO 1872 and Mux D 1868 selects the output of Jl 1873. Then Diff = 
Jl _ J0 = J n . 1 (q A l) - F(q,l) - [Jn-i(q A 0) - F(q,0)], and the output of threshold tester means 
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1876 is the survivor bit a n (q) according to Equations [4]. Trace Memory Mem_a receives 
state metric a n (q). 

Step G. Mux F 1871 selects input JO. Then Memreg 1877 = JO = J n . 
,(q*0)-F(q,0). 

5 Step H . Address addr_obs of Observation Memory Mem_obs 1 136 is set 

to read observation z„. Mux F 1871 selects as input the output of adder means 1878. 
Depending on the bit value of a n (q) the FSM will configure the ALU in one of two modes 
as follows. If a„(q) = 0, Mux E 1870 selects as input the output of latch Memreg 1877, so 
that Memreg = Memreg +/- = J n -i(q A 0) - F(q,0) +/- Za. If a n (q) = 1, Mux E 1 870 selects 

10 as input the output of latch Jl 1873, so that Memreg = Jl +/- Zn = J n -i(q A l) - F(q,l) +/- z„. 
The sign of Zn, determined by sign inverter means 167, is set negative if the MSB of state 
q is 0 and set positive otherwise. Latch Memreg 1877 then contains state metric J n (q)- 
Step I . Address addrj of State Metric Memory Mem J 1495 is set for 
writing current state metric for state q at stage n of the trellis. 

15 Step J . FSM increments state variable q. 

Step K . The current state metric is written to memory, that is, data storage 
element Memj(addrJ) receives value J n (q) in latch Memreg 1495. 

The above processing may be implemented in VLSI hardware wherein 
multiple steps are performed in parallel under FSM control and data processing clock. 

20 Table 5 summarizes the parallel processing steps and phasing of the processing clock for 
the ALU 1130 and F-Generator 1 120. Each state q at each trellis stage n requires 
computation of survivor bit a n (q), state metric J n (q), and supporting branch metric 
parameters F(q,0) and F(q,l). After initialization, the signal processing steps of the ALU 
1130 and F-Generator 1 120 are both performed in parallel in three clock cycles per state. 

25 The first row of Table 5 indicates the processing clock phasing or modulus value 0, 1, or 
2. The second row indicates at which clock phase each ALU 1 130 processing step is 
performed. The third row indicates at which clock phase each F-Generator 1 120 
processing step is performed. In Table 5, the relative timing of the processing provides at 
F-Generator 1 120 output 1 170 the values of computed branch metric parameter values 

3 0 F(q,0) and F(q, 1) precisely when needed by the ALU processing. Further there is no 

requirement for simultaneous read/write access to State Metric Memory Mem J 1 137 as 
in the preferred embodiment as the two read accesses for prior state metrics are performed 
at Step B at mod 1 and Step D at mod 2 and the one write access is performed at Step K at 
mod 0. 
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It will be apparent to a person skilled in the art that the MLSE of the 
present invention can be configured in an embodiment to support a large class of known 
soft decision generators that depend only on the partial path metrics J n °(q) and J n x (q) as 
defined by Ungerboeck in Equations [1] and [3]. The ALUs described above need only 
5 be modified to provide computed values of path metric variables JO and Jl, channel 
autocorrelation constant So, and the current state n to the soft decision generator. To see 
this let JO and Jl be represented by the notation P n °(q) and PnVl) respectively to indicate 
the dependence on state q and trellis stage n. Then at the n-th stage, the ALU makes 
available to the soft decision generator the metrics P n °(q) and P^Cq)- As described earlier 

10 the branch metrics defined for the present invention are less than the branch metrics 
defined by Ungerboeck by an amount So because of the simplification of the branch 
metric computation of the present invention in transition from Equations [3] to Equations 
[4]. Because the partial path metrics are additive with respect to the constituent branch 
metrics the pair of partial path metrics JO and Jl or alternatively P„°(q) and PnVq) are 

1 5 smaller than the corresponding partial path metrics of Ungerboeck by an amount nxs 0 . 
That is, P n °(q) = J n °(q) - nxs 0 , and P^Cq) = h\q) - nxs 0 . For some soft decision 
generators, the soft decision generators require only values JO and J 1. As an example the 
soft decision generator of Patent No. 5,502,735 calculates a soft decision for the n-th 
transmitted data symbol by b n = absolute value of [max{ J n °(q)} - max{ Jn^q)}], where 

20 the maximum is taken over all states q. By substitution, b n = absolute value of [max { 

P n °(q)} - max{ Pn^q)}] so that only values P n °(q) and V n \q), or alternatively JO and Jl are 
needed from the MLSE by the soft decision generator. 

Fig. 19 shows a flow chart for one embodiment of the present invention 
which incorporates soft decision making. Except where noted, this flow chart is also 

25 applicable for embodiments which do not use soft decision making. A data packet or 
burst, such as the GSM data burst in Fig. 1 is received, and before the processing of the 
RHS 30 and LHS 20 data is begun, parameters are input to memory and registers by a 
DSP in acts 1900 and 1905. Specifically, the RH and LD data Zn for the 58 received bits, 
along with 5 tail and dummy bits which correspond to the final and initial states, 63 in 

30 total, at an accuracy of 16 bits, are stored. The initial and final state and mask registers 
are loaded, as are the channel autocorrelation s x bits. The trellis length and, if 
appropriate, soft scale factors are loaded. 

The processing starts, act 1910, and the LH trellis is initialized. Initial 
states are initialized to 0 or -C as described above, act 1925. The branch metrics are 
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computed as per Equation [5] act 1930, and the state metrics and survivor bits are stored 
1935. Further, the difference between the partial path metrics are stored if soft decision 
making is supported. The processing continues through all the nodes at that stage of the 
trellis, 1945, 1940, and 1930. When the last state at that stage of the trellis is reached, the 
trellis stage is incremented in act 1950. At the end of the trellis, a yes to question 1955, 
the final valid state with the largest chance of occurrence is selected in act 1960. At that 
point, that is the final state, and the trace back can occur as described above, act 1965. 
When trace back finishes, the process repeats for the RH data. When processing is 
complete, the soft decision bits, if computed, are output, act 1970. 

Fig. 20 is a flow chart of one embodiment of the present invention which 
incorporates knowledge of initial and final states of the trellis. As a further refinement, 
the validity of initial and final states may also be included. A system may know that 
certain initial and final states are invalid. For example, in the GSM data burst 5 as shown 
in Fig. 1, the power ramp up and ramp down may affect the validity of the dummy bits 22 
and 32, as well as one or more of the tail bits 25 and 35. For instance, if L=5, the initial 
state for the LHS may be 1 1000 where the ones are dummy bits 22 and 32, and the zeros 
are tail bits 25 and 35. The ones and at least one zero may be invalid due to the power up 
since the autocorrelation factor s x were determined at full power. 

The trellis is initialized in act 2005. The initial and final states are read, 
2010, and the initial and final state masks are read, 2015. From this, valid initial and final 
state are determined. It is obvious to one skilled in the art that the steps listed herein may 
be done in a different order. As an example, the valid final states may be determined 
later. 

The valid initial and final states are determined in act 2020 as shown by 
the following examples. If L=5 and the initial state register contains 10010 and the initial 
state mask register contains 00000, then the only valid initial state is 10010. This is 
because the mask register indicates that all bits of the initial state register are valid. 
Alternately, if the initial state register contains 00000 and the initial state mask register 
contains 0001 1, then the four states matching OOOxx where x denotes a don't-care 
condition, that is either 0 or 1, are valid initial states. Then there are four valid initial 
states 00000, 00001, 00010, and 00011. 

The valid initial states are assigned an initial state metric of 0, and the 
invalid state are assigned an initial state metric of -C in act 2025 as described above. The 
state counter is initialized, 2030, and the branch metrics are computed, and the state 
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metric and survivor bit stored 2035. Each state q of the stage is cycled through, 2040, 
2070, and when the last state is reached 2040, the processing move to the next stage N, 
2040. When the last stage is reached, 2045, the forward trace is complete. The states q 
may be cycled through in a Gray code fashion, some of the state may be cycled through in 
a Gray code fashion while some are not. The state transitions may be limited to one bit 
changing between consecutive state, or they may be limited to two bits changing between 
consecutive states. Alternately, a limit of some other number of bits may be imposed, 
such as 3 or 4 bits. Alternately, a more conventional method of counting, such as 0000, 
0001, 0010, 001 1, 0100, or any other type of sequencing may be used. 

Next, the highest state metric of all the valid states is determined, 2050. 
This may be incorporated with the comparisons of the branch metrics for each node at the 
final stage. The trace back then begins, as the survivor bit associated with the state metric 
found in act 2050 is shifted into the final state at stage N as the LSB, 2055, resulting in a 
new state at stage N-l . The MSB that is shifted out is the last bit in the maximum 
likelihood sequence estimate. This continues in act 2060 and 280 until the last stage of 
the traceback, which is the first stage of the forward trace, is reached. 

If knowledge about the validity of the initial and final states is not known, 
that is the initial and final state masks are not known, the actual initial and final states 
may be considered the only valid states. 

Some of the following references were cited above, and all are 
incorporated by reference for all allowable purposes. 

[1] IEEE Transactions on Information Theory, vol. IT-18, pp. 363-378, 
G.D. Forney, "Maximum-likelihood sequence estimation of digital sequences in the 
presence of intersymbol interference"; 

[2] IEEE Transactions on Communications, vol. COM-22, No. 5, May 
1974, pp. 624-636, Ungerboeck, "Adaptive maximum likelihood receiver for carrier- 
modulated data-transmission systems; Proceedings of GLOBECOM'89, pp. 47.1.1- 
47.1.7 ,Nov. 1989. 

[3] J. Hagenauer and P.Hoeher, "A Viterbi Algorithm with soft-decision 
outputs and its applications"; and 

[4] Global System for Mobile Communications (GSM), Digital cellular 
telecommunications system, European Telecommunication Standard Institute (ETSI). 

While the description above provides a full and complete disclosure of the 
preferred embodiments of the present invention, various modifications, alternate 
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constructions, and equivalents will be obvious to those with skill in the art. For example, 
any memory referred to herein may be an on chip memory, memory on another chip, 
SRAM, DRAM, flip-flops, ROM, flash E-PROM, flash EE-PROM, or the like. Thus, the 
scope of the present invention is limited solely by the metes and bounds of the appended 
claims. 
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WHAT IS CLAIMED IS 

1 1 . A Maximum likelihood sequence estimator (MLSE) for estimating 

2 a sequence of transmitted symbols received over a dispersive communication channel, 

3 wherein a trellis of states and trellis paths are associated with the possible transmitted 

4 symbol sequence, said MLSE comprising: 

5 a plurality of data sources relating respectively to state transition 

6 probabilities and observed values of received data symbols; 

7 means for calculating and storing the likelihood metric and survivor bit for 

8 each state of the trellis using values from said data sources; 

9 means for determination of the final state on the maximum likelihood path 

10 in the trellis; and 

1 1 means for calculating the maximum likelihood sequence of transmitted 

12 symbols in a backward trace through the trellis using said stored survivor bits. 

1 2. The system of claim 1 further comprising: 

2 a method for computing supporting branch metric parameter calculations 

3 wherein branch metric parameters are computed recursively for a Gray coded sequence of 

4 states, wherein said recursive computation requires only a single addition operation per 

5 branch metric parameter per state, thereby substantially reducing the number of 

6 computational steps required per branch metric parameter calculation. 

1 3. The MLSE of Claim 2 wherein said branch metric parameters are 

2 pre-computed and stored in data memory prior to forward trace through the trellis, said 

3 stored branch metric parameters retrieved from memory as needed to support state metric 

4 calculations subsequently performed in forward trace through the trellis. 

1 4. The MLSE of Claim 2 wherein said branch metric parameters are 

2 computed in real time as needed for state metric calculations, and wherein the sequencing 

3 of the states is according to a Gray code for both branch metric calculations and state 

4 metric calculations, thereby achieving a substantial savings in data storage requirement. 

1 5. The MLSE of Claim 1 further including means to utilize prior 

2 knowledge about the initial trellis state to enhance MLSE estimation performance of said 

3 transmitted sequence, said performance enhancing means including: 
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4 initial state register paired with initial state mask register wherein said pair 

5 of registers define a set of valid initial states representing prior knowledge about the 

6 transmitted sequence; and 

7 means for initialization of trellis state metrics such that the MLSE Viterbi 

8 algorithm selection of the maximum likelihood path in the trellis is confined only paths 

9 having a valid initial state. 

1 6. The MLSE of Claim 1 further including means to utilize prior 

2 knowledge about the initial trellis to enhance MLSE estimation performance of said 

3 transmitted sequence, said performance enhancing means including: 

4 final state register paired with final state mask register wherein said 

5 register pair define a set of valid final states representing prior knowledge about the 

6 transmitted sequence; and 

7 means for selection of the final state of the trellis on the maximum 

8 likelihood path such that the MLSE Viterbi algorithm selection of the maximum 

9 likelihood path in the trellis is confined only to paths having a valid initial state. 

1 7. The MLSE of Claim 2 further including means to provide 

2 sufficient data for the class of soft decision generators that are dependent only on partial 

3 path metrics. 

1 8. A method of computing a maximum likelihood sequence estimate 

2 comprising: 

3 providing an initial state; 

4 providing an initial state mask comprising a plurality of bits having either 

5 a first polarity or a second plurality; and 

6 determining a plurality of valid initial states by: 

7 starting with the initial state; and 

8 substituting a don't care for each bit in the initial state which has a 

9 corresponding bit having a first polarity in the initial state mask, 

1 0 wherein the valid initial states are defined by either a one or a zero in the 

1 1 bit position having a don't care, and the same bit as the initial state in the other positions. 

1 9. The method of claim 8 wherein the initial state mask is determined 

2 by a power up characteristic of a transmitter. 
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1 10. The method of claim 9 wherein the transmitter is compliant with 

2 the Global System for Mobile standard. 

1 1 1 . A method of computing a maximum likelihood sequence estimate 

2 comprising: 

3 providing a trellis comprising a plurality of nodes corresponding to a 

4 plurality of states at a plurality of stages; 

5 providing a present state comprising a series of bits; and 

6 incrementing the present state to a next state by changing only one bit. 

1 12. The method of claim 1 1 wherein the incrementing the present state 

2 to a next state by changing only one bit is done in a Gray code. 
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MAXIMUM LIKELIHOOD SEQUENCE ESTIMATOR WHICH 
COMPUTES BRANCH METRICS IN REAL TIME 

ABSTRACT OF THE DISCLOSURE 
An improved method and apparatus for Viterbi Algorithm calculations for 
5 maximum likelihood sequence estimators in communication receivers. The calculations for 
the maximum likelihood sequence estimator, in accordance with the invention, utilizes a 
variant of the Viterbi algorithm developed by Ungerboeck in which the associated branch 
metric calculations for the trellis require the computation of a set of branch metric 
parameters. An embodiment of the present invention provides for an improved recursive 
1 0 method and apparatus for calculation of the branch metric parameters that reduces the 

number of processor clock cycles required per state metric calculation. This enables real- 
time computation of the branch metric parameters during execution of the Viterbi Algorithm. 

15 PA 3088124 vl 
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Table 1. ALU and F-Generator Processing Steps versus Clock Modulus Value 



MODULUS 
VALUE 


mod — 0 


mod = 1 


ALU steps 


A, D,G,H,J 


B, C, F, I 


F-Generator steps 


1*,3,4,5 


2*, 6 



Note *: Steps 1,2 are for initial state only 



Table 2. Required number of processing clock cycles 



FSM Phase 


No. processing clock cycles 


Explanation 


1. Forward trace init. 


0 


Reset memory 


2. Forward trace 


2 L x2xN 


2L states x 2 clock cycles per 
state x N trellis stages 


3. Backward trace init. 


0 


Concurrent with stage N of 
forward trace phase. 


4. Backward trace 


N 


N trellis stages X one clock 
cycle per stage 



Table 3. Memory requirement 



Parameters requiring 
storage (Memory module) 


Memory requirement, 
Block length N, 
channel memory L 


Memory requirement (GSM) 
N = 58, L = 5, word = 16 bits 


Branch metrics 


None 


Obits 


Correlations (Mem cor) 


L+l words 


96 bits 


State metrics (Mem j) 


2 x 2 L words 


1024 bits 


Survivor bits (Mem a) 


Nx2 L bits 


1856 bits 



Table 4. R egister contents of prior knowledge about initial and final state for GSM 







RHS data block 


Initial state 


11000 


P22P23P24P25P26* 


Initial state mask 


11100 


00000 


Final state 


P1P2P3P4P5* 


00011 


Final state mask 


00000 


00111 


Note* Midamble bit pattern 


= PlP2 P25P26 





Table 5. ALU and F-Generator Processing Steps versus Clock Modulus Value 



Modulus value 


mod = 0 


mod = 1 


mod = 2 


ALU 


A, E,K 


B, C, F, G 


D, H, I,J 


steps 




F-Generator steps 


1*,3,4,5 




2*,6 



Note *: Steps 1,2 are for initial state only 




Zso 



3to ; 



STATE 


STATE NO. 


Q 
f— 

on 

1— CQ 


0000 


0 


. 0 


0001 


1 


•] 


0010 


2 


n 
u 


0011 • 


3 


i 


0100 


U 


o 


0101 


5 


i 


0110 


6 


Q 


V i H 


7 


1 


1000 


8 


u 


1001 


9 




1010 


10 


o 


1011 


11 


1 


1100 


12 


0 


1101 


13 


1 


1110 


14 


0, 


1111 


15" 


1 



2 L STATES 





0 

o 



o 




o 











1 nl 


Qnl 


F(qn,o) 1 


F(q„,l) I 





0 


00000 


s 


F(q n ,0)-2s 5 


1 


00001 


F(q n -i,0) - 2si 


F(q n ,0)-2s 5 


2 


00011 


F(q n -i,0) - 2s 2 


F(q n ,0) - 2s 5 


3 


00010 


F(q n -i,0) + 2si 


F( qtl ,0)-2s 5 


4 


00110 


F(q n .i,0) - 2s 3 


F(q n ,0)-2s 5 


5 


00111 


F(q n -i,0)-2 Sl 


F(q n ,0) - 2s 5 


6 


00101 


FCqn-i.O) + 2s 2 


F(q n ,0) - 2s 5 


7 


00100 


F(q„-i,0) + 2s! 


F(q n ,0) - 2s 5 


8 


01100 


F(q„.i,0) - 2s 4 


F(q„,0)-2s 5 


9 


01101 


F(q n -i,0)-2si 


F(q n ,0) - 2s 5 


10 


01111 


F(q n -i,0) - 2s 2 


F(q n ,0)-2s 5 


11 


omo 


F(q„-i,0) + 2si 


F(q n ,0)-2s 5 


12 


01010 


F(q n -i,0) + 2s 3 


F(q n ,0)-2s 5 


13 


01011 


F(q n -i,0) - 2si 


F(q n ,0) - 2s 5 


14 


01001 


F(q n .i,0) + 2s 2 


F(q n ,0)-2s 5 


15 


01000 


F(q„-i,0) + 2si 


F(q n ,0) - 2s 5 


16 


10000 


-s 


F(q n ,0) + 2s 5 


17 


10001 


F(q n -i,0) + 2si 


F(q n ,0) + 2s 5 


18 


10011 


F(q n -i,0) + 2s 2 


F(q n ,0) + 2s 5 


19 


10010 


F(q n -i,0) - 2si 


F(q n ,0) + 2s 5 


20 


10110 


F(q n -i,0) + 2s 3 


F(q n ,0) + 2s 5 


21 


10111 


F(q n -i,0) + 2si 


F(q n ,0) + 2s 5 


22 


10101 


F(q n .i,0) - 2s 2 


F(q n ,0) + 2s 5 


23 


10100 


F(q„.i,0) - 2 Sl 


F(q n ,0) + 2s 5 


24 


11100 


F(q n -i,0) + 2s 4 


F(q n ,0) + 2s 5 


25 


11101 


F(q„-!,0) + 2 Sl 


F(q n ,0) + 2s 5 


26 


11111 


F(q n -i,0) + 2s 2 


F(q n ,0) + 2s 5 


27 


11110 


F(q n .i,0)-2 Sl 


F(q n ,0) + 2s 5 


28 


11010 


F(q„-i,0)-2s 3 


F(q n ,0) + 2s 5 


29 


11011 


F(q n -i,0) + 2si 


F(q n ,0) + 2s 5 


30 


11001 


F(q n -i,0)-2s 2 


F(q n ,0) + 2s 5 


31 


11000 


F(q„- b O) - 2 Sl 


F(q n ,0) + 2s 5 



Fig. 7 
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Attorney Docket No.: 20669-000300US 



DECLARATION AND POWER OF ATTORNEY 



As a below named inventor, I declare that: 



My residence, post office address and citizenship are as stated below next to my name; I believe I am the original, first and sole 
inventor (if only one name is listed below) or an original, first and joint inventor (if plural inventors are named below) of the subject 
matter which is claimed and for which a patent is sought on the invention entitled: MAXIMUM LIKELIHOOD SEQUENCE 
ESTIMATOR WHICH COMPUTES BRANCH METRICS IN REAL TIME the specification of which JL_ is attached hereto 
or was filed on as Application No. and was amended on (if applicable). 

I have reviewed and understand the contents of the above identified specification, including the claims, as amended by any 
amendment referred to above. I acknowledge the duty to disclose information which is material to patentability as defined in Title 37, 
Code of Federal Regulations, Section 1.56. I claim foreign priority benefits under Title 35, United States Code, Section 119 of any 
foreign application(s) for patent or inventor's certificate listed below and have also identified below any foreign application for patent 
or inventor's certificate having a filing date before that of the application on which priority is claimed. 



Prior Foreig n Application(s) 



Country 


Application No. 


Date of Filing 


Priority Claimed Under 
35 USC 119 











"I hereby claim the benefit under Title 35, United States Code § 1 19(e) of any United States provisional application(s) listed below: 



Application No. 


Filing Date 


60/147,506 


08/09/99 



I claim the benefit under Title 35, United States Code, Section 120 of any United States application(s) listed below and, insofar as the 
subject matter of each of the claims of this application is not disclosed in the prior United States application in the manner provided by 
=the first paragraph of Title 35, United States Code, Section 1 12, 1 acknowledge the duty to disclose material information as defined in 
=Title 37, Code of Federal Regulations, Section 1.56 which occurred between the filing date of the prior application and the national or 
=PCT international filing date of this application: 



Application No. 


Date of Filing 


Status 









POWER OF ATTORNEY: As a named inventor, I hereby appoint the following attorney(s) and/or agent(s) to prosecute this 
application and transact all business in the Patent and Trademark Office connected therewith. 



J. Matthew Zigmant, Reg. No. 44,005 
Gary T. Aka, Reg. No. 29,038 



Send Correspondence to: 


Direct Telephone Calls to: 


J. Matthew Zigmant 


(Name, Reg. No., 


Telephone No.) 


TOWNSEND and TOWNSEND and CREW LLP 


Name: 


J. Matthew Zigmant 


Two Embarcadero Center, 8 th Floor 


Reg. No.: 


44,005 


San Francisco, California 94111-3834 


Telephone: 


650-326-2400 
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Full Name of 




First Name: 


Middle Name or Initial: 


Inventor 1: 


POPE 


STEPHEN 


P. 




Residence & 


City: 


State/Foreign Country: 


Country of Citizenship: 


Citizenship: 


Berkeley 


California 


United States 




Post Office 


Post Office Address: 


City: 


State/Country: 


Postal Code: 


Address: 


2128 Acton Street 


Berkeley 


California 


94702 



I further declare that all statements made herein of my own knowledge are true and that all statements made on information and belief 
are believed to be true; and further that these statements were made with the knowledge that willful false statements and the like so 
made are punishable by fine or imprisonment, or both, under Section 1001 of Title 18 of the United States Code, and that such willful 
false statements may jeopardize the validity of the application or any patent issuing thereon. 



Signature of Inventor 1 



Stephen P. Pope 
Date 
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